Papers with attention-based methods
Graph Attention Network with Memory Fusion for Aspect-level Sentiment Analysis (2020.aacl-main)
Copied to clipboard
| Challenge: | Recent studies ignored the syntactic relationship between the aspect and its corresponding context words, leading the model to focus on syntaktically unrelated words mistakenly. |
| Approach: | They propose to extend the graph convolutional network by assigning different weights to edges of connected words. |
| Outcome: | The proposed method can improve on five datasets showing that it learns and exploits multiword relations and draws different weights of words to improve performance. |
Zero-Shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens (N18-1)
Copied to clipboard
| Challenge: | Recent work has used attention weights to visualize the focus of neural models in input data. |
| Approach: | They propose to use attention-based visualization techniques to infer token-level labels from a network trained only on sentence-level labeling. |
| Outcome: | The proposed approach outperforms gradient-based methods on four datasets and is expected to outperfect supervised methods. |
DET: A Dual-Encoding Transformer for Relational Graph Embedding (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to graph representation only consider the local neighbors, sacrificing the Transformer’s ability to attend to elements at any distance. |
| Approach: | They propose a dual-encoding Transformer architecture that uses a structural encoder and a semantic encoder to seek for semantically relevant nodes. |
| Outcome: | The proposed architecture achieves superior performance compared to state-of-the-art attention-based methods on complex relational graphs like KGs and citation networks. |
CAN: Constrained Attention Networks for Multi-Aspect Sentiment Analysis (D19-1)
Copied to clipboard
| Challenge: | Existing methods for aspect-specific sentiment classification are noisy and downgraded performance. |
| Approach: | They propose a constrained attention network to regularize attention for multi-aspect sentiment analysis by orthogonal regularization on multiple aspects and sparse regularization for each single aspect. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on two public datasets and extends to multi-task settings. |
Hallucination Detection in LLMs Using Spectral Features of Attention Maps (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performance across tasks but remain prone to hallucinations. |
| Approach: | They propose a method that uses attention maps to detect hallucinations . they propose to use top-k eigenvalues of the attention maps as input to probes . |
| Outcome: | The proposed method achieves state-of-the-art hallucination detection performance among attention-based methods. |
DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metrics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-agent debates can improve the accuracy of Large Language Models by having multiple agents discuss solutions over several rounds of debate. |
| Approach: | a debate framework that uses uncertainty metrics to assess agent confidence is proposed . the framework uses textual prompts or a modified attention mechanism that adjusts token weights . |
| Outcome: | The proposed framework assesses agent confidence using uncertainty metrics . the framework is available at https://github.com/lukeyoffe/debunc. |